Requirements Metrics: Scaling Up

Establishing the relative value of results within a field of study contributes to advancement of that field. In order to compare across large numbers of results together, the methods and metrics used must be scaled up from existing studies where the number of subjects or cases is small. This scaling provides specific challenges, for example, metrics used in small studies are often shaped by factors of the local environment. Standardization is required in order to enable aggregation of data across multiple distributed environments. Further, standardization of metrics is both non-trivial and insufficient. Complex linguistic factors must be accounted for in order to maximize consistency of metric interpretation and use, and a number of other issues must be addressed to ensure that the metrics are interesting as well as practical. This position paper elaborates these issues, sets forth criteria for benchmark-friendly metrics, and proposes a community activity designed to establish a foundational set of requirements benchmark metrics.